Tools help us accomplish our daily tasks. A parent uses an electric or gas range, pots and pans, and spoons, spatulas and other implements to cook a meal for his or her family. A carpenter uses planes, saws, hammers and levels to cut pieces of wood to size and fasten them in. A mechanic uses wrenches, gear pullers, jacks, lifts and other tools to disassemble and reassembly a car engine. A gardener uses rakes, hoes, clippers and other implements to help nurture plants.
Just as in the real world, tools are also important in virtual realty, augmented reality and computer graphic simulations. Therefore, much work has been done in the past to develop efficient ways for a human user to control virtual tools displayed on a screen. In some cases, nearly the same controls a human user will operate in the real world can be used to control a computer graphics simulation of the same tool.
For example, an aircraft simulator used to train pilots often resembles nearly exactly a cockpit of an airplane. Large display screens are used to display what a pilot would see outside the window as if her or she were flying an actual airplane. The pilot in training operates the same controls found in a real cockpit to control the virtual aircraft simulation.
As another example, surgeons are sometimes trained using computer simulators that simulate the way the human body responds to surgical procedures. In such surgical simulations, a virtual scalpel may be used to cut tissue, a virtual suction device may be used to suction away blood and debris, and a virtual suture may be used to sew up the simulated wound. Such simulations allow surgeons to try new surgical techniques—oftentimes using real world controls that very closely resemble the implements they will use in the operating room—without risking the health and well being of actual human patients. Although there is no substitute for actual real world experience, such simulations can be useful as a first step toward training new surgeons.
Although designers of expensive aircraft, surgical and other simulators have the luxury of advanced, authentic controls and other input devices, designers of video game systems and other consumer based computer systems are often presented with a different set of challenges. For example, the conventional home video game platform, personal computer or handheld video game playing platform often has a relatively limited set of standard, general-purpose input devices the human player can use to manipulate video game action. Similarly, the computer keyboard and mouse of a conventional personal computer are generally used to control nearly all game functions of video games played on personal computers. Serious PC gamers will sometimes invest in a joystick peripheral device to allow more advanced directional control of certain games. However, such additional peripheral devices become impractical when gamers attempt to play games on portable platforms such as portable video gaming systems, cellular telephones with graphics capabilities, personal digital assistants, pocket PCs, and a host of other devices. In those contexts, the control interface that is available with the device out of the box is typically the one most players end up using for nearly all of their interactions. A challenge is to make maximum use of such input devices to create user-friendly, effective graphical user interfaces.
But while designers of video games, virtual and augmented reality based systems, and other virtual computer-generated environments may have certain limitations, they also have an additional degree of freedom. Just like in the real world, a character within a video game or other computer-generated environment may also use tools to accomplish certain virtual goals. But because computer generated environments are virtual, developers are not limited to real world tools. They can create new or fanciful tools not possible in the real world.
For example, the main character Link in Nintendo's highly successful Legend of Zelda video game series uses a sword, a musical instrument called an ocarina, a “windwaker” stick, a bow and arrow, a boomerang, a telescope, a fishing rod, and various other tools to accomplish certain results within the games. Some of these tools resemble real world objects, but within the game they often have magical properties. For example, Link can travel through time and otherwise magically manipulate his environment by playing certain songs on his ocarina.
Another interesting example is the “Poltergust 4000” vacuum cleaner strapped to the game character Luigi's back in Nintendo's renowned “Luigi's Mansion” video game. While this virtual vacuum cleaner has some resemblance to the kind of vacuum cleaner you clean your carpets with, it has amazing properties no real world vacuum cleaner would ever exhibit.
The human game player can operate Luigi's “Poltergust 4000” vacuum cleaner just like a real vacuum cleaner to “suck” (suction mode) or “blow” (exhaust mode). In the suction mode, the virtual vacuum cleaner sucks up objects within Luigi's virtual environment including but not limited to ghosts, fire spirits, water, ice cubes and balls. In the exhaust mode, the vacuum cleaner can exhaust some of these vacuumed-up items in fanciful ways. For example, if a fire spirit was the last item Luigi's vacuum cleaner sucked up, then operating Luigi's vacuum cleaner in the exhaust mode transforms the vacuum cleaner into a flame thrower. Similarly, if the last item the game player caused Luigi's vacuum player to suck up was water from a fountain, then Luigi's vacuum cleaner will in the exhaust mode act as a fire hose that emits a virtual water spray. This creative way of enhancing the functionality of an everyday appliance with magical or extraordinary properties has found great favor among millions of video game players throughout the world.
Despite the limitations in low cost video game and other consumer systems, there is an interesting relationship between the input devices the human game player uses to control a video game or other virtual environment, and the tools within the video game or other virtual environment used to accomplish objectives. In video games such as the Legend of Zelda and Luigi's Mansion, the human player uses buttons and/or joy sticks on a handheld controller (e.g., the “C stick”) to control the operation of the tools the game characters use within the game. For example, in the Luigi's Mansion game, the human player can control whether Luigi's virtual “Poltergust 4000” vacuum cleaner is in the suction mode, the exhaust mode, a flashlight mode, or no mode by manipulating buttons and/or joysticks on the Nintendo GameCube handheld controller. The handheld controller functions as a control panel for Luigi's virtual vacuum cleaner, allowing the human video game player to control where the “Poltergeist 4000” vacuum cleaner is aiming and what mode it is operating in. The same joysticks and buttons would be used to control direction, acceleration, turning radius and other attributes of a vehicle in a driving or flight simulator game.
Video game and computer graphics designers are constantly searching for new and innovative ways to make interaction with virtual, simulated and augmented reality worlds more efficient, cost-effective and enjoyable. Recently, touch screen based user interfaces have been introduce in the context of video game systems, banking terminals and other computer devices as a way to interact more efficiently with a graphical-based computer system. However, further improvements are possible and desirable.
The technology herein uses a 3D graphical computer interface to enhance the function of a physical object used to interact with a video game, computer simulation or any other graphical presentation. The object a human user holds in his or her hands may be relatively static or non-electronic, and yet may—through interaction with a specialized user interface provided via a multimedia (e.g., audio and/or graphical video) display presentation—appear to possess enhanced functional characteristics and properties. An ordinary handheld object may, through interaction with a computer graphical interface, appear, for example, to take on extraordinary or even magical virtual tool characteristics.
In one specific exemplary illustrative non-limiting implementation, the stylus used to interact with a touch screen based video game interface may, through simulation of enhanced functions on a computer graphics screen, appear to take on characteristics of a virtual vacuum or suction tool. In this exemplary illustrative non-limiting implementation, the stylus itself may be a conventional elongated stick-like implement such as a piece of inert hard plastic. Such a stylus in one exemplary illustrative non-limiting implementation has no internal cavity, no electronics, and substantially no mechanical function other than to indicate touching positions on a touch pad or touch screen. An exemplary illustrative non-limiting implementation of the technology herein uses computer graphics and computer-generated sound to animate interaction with such a conventional stylus to provide suction and exhaust properties of a virtual suction tool.
For example, unlike a real suction tool which may have relatively limited ability to pick up and put down real world objects in terms of dimensions, object characteristics and the like, the virtual suction tool provided by one exemplary illustrative non-limiting implementation of the technology herein can have fanciful object manipulation capabilities such as infinite capacity to suck up objects of all sorts. In a puzzle game for example, the virtual suction tool may be able to suck up any number of bubbles, blocks or other objects within the game. As the human user moves the tip of the virtual suction tool over the objects, each object may disappear in a way that creates an impression that the stylus is sucking the object up into an internal cavity or reservoir within the stylus. For example, as the object is being virtually suctioned, portions of the object closest to the tool may deform on the screen to make it appear as if the object has been partially sucked into the virtual vacuum tool. Once the stylus virtually contains the entire object that has been sucked up, the object may disappear from the display.
In a further exemplary non-limiting illustrative implementation, a noise, such as a sucking or whooshing noise, may accompany each capture of an object, so that the player feels as if the stylus has actually sucked the object from the screen. Similarly, an exhaust sound can accompany virtual “exhaust” of an object back into the virtual environment. The human player hearing a computer-generated suction sound creates the impression that the virtual suction tool stylus or other object has sucked up the object and now “contains” the object. The stylus or other object may virtually “store” any number of digital objects, the capacity being limited only by the game design.
In example illustrative non-limiting implementations, the human player may move the stylus to a different position in the game and expel the same (or transformed) object(s) back into the game environment. Accompanying exhaust sounds and graphical effects that make it appear as if the stylus is actually expelling them into the game environment. Alternatively or in combination, the object may be transformed by the virtual suction device (e.g., a red block may be transformed into fire) in the process of being virtually exhausted or expelled into the game environment.
The technology herein is thus capable of transforming an everyday common physical object such as an inert stylus into a device that may but does not necessarily need to exist in the real world. Exemplary illustrative non-limiting implementations thus provide methods and apparatus for a player to cross the boundary between the virtual world and the real world. Through displayed virtual world images and sounds, the player gets the impression that a virtual object can be brought into and manipulated in the real world. He similarly has the feeling that the physical object he is holding in his hand has virtual tool properties, functions and/or characteristics defining the manner in which it interacts with the virtual world. For example, in the case of a stylus being used as a virtual suction tool, the player may see the object distort and shrink as it is “sucked into” the stylus—even though the stylus is incapable of applying suction in the real world, is not connected to a vacuum pump or vacuum reservoir, and has no internal reservoir or other capacity for containing objects or material.
Accompanied by a suction-like sound, the player is given the impression that the plastic stylus has actually pulled and captured an object from the virtual world, bringing it into his world. If the player decides to empty the contents of the virtual suction tool back into the virtual world, an animation of the object being expelled from the stylus accompanied by a distinctive “exhaust” sound gives the player the impression that the object (or a transformed version of the object) has been returned from the real world back into the virtual world.
Particularly advantageous exemplary illustrative non-limiting implementations make use of a touch screen to provide the boundary between the real world and the 3D virtual world. A stylus, finger or other object interacting with a touch screen provides a convenient way to indicate position in the virtual world where the interaction is to occur. The fact that the stylus, finger or other object is directly in contact with the touch screen allows virtual world graphics at the point of contact to appear to be interacting directly with the physical object. Such an “Alice Through the Looking Glass” property of the virtual suction tool is also highly intuitive and natural and easy to use.
Other alternative arrangements using other pointing or position indicating arrangements, with or without distance between the object and the display, can also be provided. Thus, the physical object in one exemplary illustrative non-limiting implementation can be a touch screen stylus, but the physical object can be any object capable of interacting with a 3D or other graphics or other display system including but not limited to remote control devices, specially designed objects that have physical appearance of tools or weapons, etc. Since the physical object used for interaction is given virtual or simulated characteristics, capabilities and/or functions, the resulting virtual tool can differ from and/or exceed functions, capabilities and/or characteristics of actual physical objects existing in the real world. For example, a virtual suction tool can have properties that differ from and/or exceed the properties of a real world suction tool. An inexpensive, inert or other piece of plastic or other material can serve as a variety of different tools for a variety of games, each tool resembling, simulating or exceeding the bounds of any real world equivalent. The types and functions of each virtual tool are limited only by the imagination of the game developer. Because the game player has direct control over the virtual objects and can manipulate them seemingly in the real world, the player gains a high degree of control and a very interesting and engaging gaming experience. In a gaming context, the player feels as if he is directly manipulating the game object and so has a close connection to the game experience.